I read rapidsai/build-planning#118 to find out why preferring the system library over the one from a wheel is "a good idea". But that issue doesn't really explain it, except it mentions that for most users who don't care about packaging the plan is to use the library from the wheel. Which makes me think "shouldn't we be doing that as devs as well?" - kinda dogfooding, kinda making sure this is the most comfortable path, etc

I think the change itself is uncontroversial (the magic of using the env variable once it is set is presumably hidden away somewhere and tested?), so for me it is about understanding "why make this change".

It looks like the failing tests are all related to dask. I know Dante and friends were talking about there being some issue with dask, so maybe we wait and see? Or someone who knows can chime in :D

Thanks for the thorough review @betatim . Let me try to address these.

why preferring the system library over the one from a wheel is "a good idea". But that issue doesn't really explain it,

The main goal of rapidsai/build-planning#118 is to change our library-loading logic across RAPIDS such that if you installed wheels, at runtime by default you get the wheel-provided library, NOT any system-provided one that you might have laying around).

Doing this makes it safer to use pip install cuml-cu12, because you can use pip to manage the UCX dependency. Like pip install cuml-cu12 'libucx>=1.15.0,<1.16'. That's helpful for cases where, for example, you have permissions to pip install things but not to change system libraries (like apt-get install, yum install, etc.).

In RAPIDS devcontainers, we want to prefer the system installation because we're producing specific images for different UCX versions. See the ucx1.17.0 in this tag:

And for other reasons related to how the devcontainers are designed (see next response below).

shouldn't we be doing that as devs as well?" - kinda dogfooding

Yes there would be some benefits to just using the libucx-cu{11,12} wheels to provide the UCX libraries in the RAPIDS devcontainers, but there are other tradeoffs there. For example, the devcontainers are designed to make it easy to do C++-only development (e.g. compiling libcuml++ and testing it) without needing to involve Python or pip at all. In that case, "get the UCX libraries from wheels" might be challenging.

But that design discussion is not specific just to UCX, and is bigger than this series of PRs. That longer-term discussion is happening here: rapidsai/build-planning#119

the magic of using the env variable once it is set is presumably hidden away somewhere and tested

Yes. For the specific case of UCX (libucx-cu{11,12} wheels):

• change that introduced the env variable and change in library-loading preference: prefer wheel-provided libraries, use RTLD_LOCAL ucx-wheels#13
• how that was tested: prefer wheel-provided libraries, use RTLD_LOCAL ucx-wheels#13 (comment)

Thanks a lot for taking the time to explain. Sounds reasonable to me.

@dantegd any thoughts on the failing (dask) tests? Should we wait for that to get resolved before merging this?

I just merged latest branch-25.02 in here to kick off a full new CI run, to see if that'd help resolve the issues we were seeing... and now I see something different.

All tests and the docs builds are failing with this runtime error:

ConftestImportFailure: ModuleNotFoundError: No module named 'cuda.bindings' (from /__w/cuml/cuml/python/cuml/cuml/tests/conftest.py)

/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/config/__init__.py:331: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
ConftestImportFailure: ModuleNotFoundError: No module named 'cuda.bindings' (from /__w/cuml/cuml/python/cuml/cuml/tests/conftest.py)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config = pluginmanager.hook.pytest_cmdline_parse(
ImportError while loading conftest '/__w/cuml/cuml/python/cuml/cuml/tests/conftest.py'.
../conftest.py:17: in <module>
    from cuml.testing.utils import create_synthetic_dataset
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/__init__.py:17: in <module>
    from cuml.internals.base import Base, UniversalBase
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/__init__.py:18: in <module>
    from cuml.internals.base_helpers import BaseMetaClass, _tags_class_and_instance
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/base_helpers.py:20: in <module>
    from cuml.internals.api_decorators import (
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/api_decorators.py:24: in <module>
    from cuml.internals import input_utils as iu
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/input_utils.py:20: in <module>
    from cuml.internals.array import CumlArray
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/array.py:21: in <module>
    from cuml.internals.global_settings import GlobalSettings
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/global_settings.py:20: in <module>
    from cuml.internals.device_type import DeviceType
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/device_type.py:19: in <module>
    from cuml.internals.mem_type import MemoryType
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/mem_type.py:22: in <module>
    cudf = gpu_only_import("cudf")
/opt/conda/envs/test/lib/python3.10/site-packages/cuml/internals/safe_imports.py:362: in gpu_only_import
    return importlib.import_module(module)
/opt/conda/envs/test/lib/python3.10/site-packages/cudf/__init__.py:19: in <module>
    _setup_numba()
/opt/conda/envs/test/lib/python3.10/site-packages/cudf/utils/_numba.py:121: in _setup_numba
    shim_ptx_cuda_version = _get_cuda_build_version()
/opt/conda/envs/test/lib/python3.10/site-packages/cudf/utils/_numba.py:16: in _get_cuda_build_version
    from cudf._lib import strings_udf
/opt/conda/envs/test/lib/python3.10/site-packages/cudf/_lib/__init__.py:4: in <module>
    from . import (
copying.pyx:1: in init cudf._lib.copying
    ???
column.pyx:1: in init cudf._lib.column
    ???
/opt/conda/envs/test/lib/python3.10/site-packages/rmm/__init__.py:17: in <module>
    from rmm import mr
/opt/conda/envs/test/lib/python3.10/site-packages/rmm/mr.py:14: in <module>
    from rmm.pylibrmm.memory_resource import (
/opt/conda/envs/test/lib/python3.10/site-packages/rmm/pylibrmm/__init__.py:15: in <module>
    from .device_buffer import DeviceBuffer
device_buffer.pyx:1: in init rmm.pylibrmm.device_buffer
    ???
E   ModuleNotFoundError: No module named 'cuda.bindings'

(build link)

Those jobs are getting cuda-python 11.8.3

cuda-python               11.8.3          py310hda4ad70_3    conda-forge

Seeing cudf and rmm in the stacktrace is making me think the root cause is that cuml didn't yet get updates like these:

• Remove upper bounds on cuda-python to allow 12.6.2 and 11.8.5 cudf#17326
• Remove upper bounds on cuda-python to allow 12.6.2 and 11.8.5 rmm#1729

That should be resolved by merging #6157, which is blocked by the previously-mentioned Dask issues.

That should be resolved by merging #6157, which is blocked by the previously-mentioned Dask issues.

It's now #6170 that has CI fixes in it, and that needs to be merged before we can merge this and other PRs targeting 25.02.

@jameslamb unfortunately there's still a devcontainer pip issue that we are trying to solve in #6170, but the cuml dask failures should be green now though

/merge

Thanks for coming back and updating this @bdice !